I. Introduction

In the three previous experiments, we have shown that people infer trustworthiness from convergence - the extent to which different informants agree. Greater convergence makes (i) an information to be perceived as more accurate and (ii) the informants as more competent - that is, given certain context.

Some aspects of that context we held constant; others we varied to evaluate how they alter the effect of convergence. We held constant that participants did not have any priors whatsoever - not on the information they received (they knew nothing about the games that were played) nor about the informants (the players who played the games). We manipulated informants’ independence. We tested two cases of independence, by varying the baseline of what dependent meant: In experiment two, dependent informants had discussed “at great length” before making their estimates. In experiment three, dependent informants shared the same conflict of interest of making a certain estimate.

In this first series of experiments (experiments one, two and three) we tested inferences from convergence in a numerical choice setting: Participants saw (fictive) players’ estimates that would lie with 1000 and 2000. The degree of convergence varied by the distance between estimates.

In a second series of experiments (experiments four, five and six), we test inferences from convergence in a categorical choice setting. In the categorical scenario, the fictive players make choices on a set of response options (i.e. categories). Convergence varies by the ratio of people agreeing on an option. Experiment four and five can be considered robustness checks as to whether the results of the first series hold in a categorical choice setting. Experiment six tests a new context factor: the number of choice options. In experiments four and five, scenarios always involved three choice options. Here, we will vary between three and ten options. The design we use to manipulate convergence is otherwise identical to experiment four (players playing games).

II. Data collection

No data has been collected yet.

III. Design

Participants see eight situtations in which three advisors give investment recommendations. They are then asked to judge one advisor’s accuracy and competence. We will manipulate two factors: convergence (four levels, within participants) and options (two levels, between participants).

Introduction for participants: “To be able to understand the task, please read the following instructions carefully: Some people are playing games in which they have to select the correct answer among three answers. You will see the results of several of these games. Each game is different, with different solutions and involving different players. All players answer independently of each other. At first, you have no idea how competent each individual player is: they might be completely at chance, or be very good at the task. It’s also possible that some players are really good while others are really bad. Some games might be difficult while others are easy. Your task will be to evaluate the performance of one of the players based on what everyone’s answers are.”

An example of a stimulus for the majority condition

Convergence. Convergence varies by the ratio of players choosing the same response as the focal player (i.e. the one that participants evaluate). The levels of convergence are: (i) consensus, where all three players pick the same option [coded value = 3]; (ii) majority, where either the third or second player picks the same option as the first player [coded value = 2]; (iii) dissensus, where all three players pick different options [coded value = 1]; (iv) majority against the focal player’s estimate, where the second and third player pick the same option, but one that is different from the first player’s choice [coded value = 0]. In our analysis, we treat convergence as a continuous variable, assigning the values in squared parenthesis.

We manipulate convergence within participants. All participants see all four conditions of convergence, with two stimuli (i.e. game results) per condition. Each participant therefore sees eight stimuli in total (4 convergence levels x 2 stimuli) .

Stimuli for 3 options condition by levels of convergence
Level Version a) Version b)
opposing majority (0)
dissensus (1)
majority (2)
consensus (3)

Options. Independence has two levels: (i) ‘3’ and (ii) ‘10’.

We manipulate independence between participants, i.e. each participant gets assigned to either the ‘3’ or the ‘10’ condition. All stimuli that participants see in their respective condition will involve the same number of choice options.

Example of a consensus stimulus for the two ‘Option’ conditions
Convergence Options: 3 Options: 10
consensus (3)

As outcome variables, we will measure people’s perceived accuracy and competence of player one.

Accuracy. We ask participants “What do you think is the probability of player 1 being correct?”. Participants answer with a slider from 0 to 100.

Competence. We ask participants “How competent do you think player 1 is in games like these?” Participants answer on a 7-point Likert scale (from “not competent at all” to “extremely competent”).

IV. Hypotheses

Considering only 3 options condition of the experiment at hand, we expect the findings of experiment four to replicate. In experiment four, players would choose among 3 options.

H1a: Participants perceive an estimate of an independent informant as more accurate the more it converges with the estimates of other informants, given three choice options.

To test this hypothesis, we only consider participants assigned to the 3 options condition.

We use a linear mixed effect model with random intercept and random slope per participant. Should this model yield convergence issues, we will use a model with random intercept only.

In all our models we treat convergence as a continuous variable. We will, however, include robustness checks where we treat convergence as a categorical variable, allowing to inspect difference between different levels.

# models for accuracy

# random intercept and slope by participants
model_accuracy <- lmer(accuracy ~ convergence + (1 + convergence | id), 
                       data = data %>% filter(options == "3"))

# in case of non-convergence: random intercept by participants only
alt_model_accuracy <- lmer(accuracy ~ convergence + (1 | id), 
                           data = data %>% filter(options == "3"))

H1b: Participants perceive an independent informant as more competent the more their estimate converges with the estimates of other informants, given three choice options.

To test this hypothesis, we only consider participants assigned to the 3 options condition.

We will proceed in the same way for competence as we did for accuracy above.

# models for competence

# random intercept and slope by participants
model_competence <- lmer(competence ~ convergence + 
                           (1 + convergence | id), 
                         data = data %>% filter(options == "3"))

# in case of non-convergence: random intercept by participants only
alt_model_competence <- lmer(competence ~ convergence + (1 | id), 
                             data = data %>% filter(options == "3"))

How about a context in which informants (i.e. the fictive players) could choose not among 3, but among 10 options? In line with our findings from experiment three, we predict that effects of convergence are more positive when there are more choice options:

H2a: The effect of convergence on accuracy (H1a) is more positive in a context where informants can choose among ten response options compared to when they can choose among only three.

To test this hypothesis, we consider the full data.

The resulting estimate of the interaction term will provide the test for our hypothesis.

# models for accuracy

# random intercept and slope by participants
model_accuracy <- lmer(accuracy ~ convergence + options + 
                            options*convergence + (1 + convergence | id), 
                       data = data)

# in case of non-convergence: random intercept by participants only
alt_model_accuracy <- lmer(accuracy ~ convergence + options + 
                            options*convergence + (1 | id), 
                           data = data)

H2b: The effect of convergence on competence (H1b) is more positive in a context where informants can choose among ten response options compared to when they can choose among only three.

To test this hypothesis, we consider the full data.

The resulting estimate of the interaction term will provide the test for our hypothesis.

# models for competence

# random intercept and slope by participants
model_competence <- lmer(competence ~ convergence + options + 
                            options*convergence + (1 + convergence | id), 
                       data = data)

# in case of non-convergence: random intercept by participants only
alt_model_competence <- lmer(competence ~ convergence + options + 
                            options*convergence + (1 | id), 
                           data = data)

Robustness checks

Convergence as categorical variable

In the models above, we treated convergence as a continuous variable. Based on the different levels, we will build a categorical variable, convergence_categorical.

# make a categorical variable from `convergence`
data <- data %>% 
  mutate(convergence_categorical = recode_factor(convergence, 
                                                 `0` = "opposing majority", 
                                                 `1` = "divergence", 
                                                 `2` = "majority", 
                                                 `3` = "consensus",
                                                 .default = NA_character_)
         )

levels(data$convergence_categorical)
## [1] "opposing majority" "divergence"        "majority"         
## [4] "consensus"

We run the same models outlined in the hypotheses section, but replacing convergence with convergence_categorical. This also allows us to inspect heterogeneity in differences between levels (with respect to the baseline, i.e. “opposing majority”).

Alternative stimuli with greater distance

We kept the same distance between 10 and 3. That’s conservative - people might think that other options are not relevant and hence don’t make a difference between 3 and 10 options. We therefore run the hypotheses set two comparing 3 to a version of 10 with larger distance.

Pick an example stimulus as earlier for illustrating choice options. See all stimuli in appendix.

Exclusions

We will exclude participants failing (i.e. participants not answering the question or writing anything that does not at least resemble “I pay attention”) the following attention check:

Imagine you are playing video games with a friend and at some point your friend says: “I don’t want to play this game anymore! To make sure that you read the instructions, please write the three following words”I pay attention” in the box below. I really dislike this game, it’s the most overrated game ever. Do you agree with your friend?

Power analysis

We ran a power simulation to inform our choice of sample size. All assumptions and details on the procedure can be found in the power_Exp5.Rmd document. We ran two different power analyses, one for each outcome variable. We set the power threshold for our experiment to 90%.

The power simulation for accuracy suggested that for 80 participants, we would have a power of at least 90% for the interaction effect. The simulation for competence suggested that with already 40 participants, we would detect an interaction, but only with 60 participants we also detect an effect of convergence.

However, due to uncertainty about our assumptions and because we are anticipating failed attention checks, we will recruit a sample of 200 participants.

Appendix

Stimuli for 10 options condition by levels of convergence
Level Version a) Version b)
opposing majority (0)
dissensus (1)
majority (2)
consensus (10)
Alternative stimuli for 10 options condition by levels of convergence
Level Version a) Version b)
opposing majority (0)
dissensus (1)
majority (2)
consensus (10)